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Abstract. In this paper we investigate a finite temperature generalization of survey 
propagation, by applying it to the problem of finite temperature decoding of a 
biased finite connectivity Sourlas code for temperatures lower than the Nishimori 
temperature. We observe that the result is a shift of the location of the dynamical 
critical channel noise to larger values than the corresponding dynamical transition for 
belief propagation, as suggested recently by Migliorini and Saad for LDPC codes. We 
show how the finite temperature 1-RSB SP gives accurate results in the regime where 
competing approaches fail to converge or fail to recover the retrieval state. 



PACS numbers: 89.70+c, 89.90+n, 05.50+q 
1. Introduction 

In the recent past, there have been a number of developments in the field of information 
processing and computer science that were triggered by insights originating from 
statistical mechanics. A number of them was initiated after the discovery of the 
close relationship between the sum-product algorithm or loopy belief propagation pQ 
in computer science on the one hand, and the Bethe approximation, the cavity method 
and the replica method in statistical mechanics on the other hand. [3 El- The 
belief propagation algorithm is a method to compute approximations of the microstate 
probability distribution or the corresponding marginals of variables that interact on a 
given graph. This objective arises in various statistical inference problems with which 
the machine learning community is concerned, such as stereo vision optimization 
problems j3], image restoration [6 , digital zoom and shape-from-shading [7j, error 
correcting codes for telecommunication [HUH], human movement modelling JO], protein 
folding [11] evaluating postitions in the game of Go ^2]- On the other hand, in the 
field of statistical mechanics, the main interest is the macroscopic behaviour of systems, 
and thus the objective is the calculation of macroscopic observables describing ensemble 
averages over graphs distributed according to a given measure. In the limit of infinite 
system size, the macroscopic observables for a given instance of a graph coincides with 
the ensemble average in many models of interest. 



Survey propagation at finite temperature: application to a Sourlas code as a toy model2 

The cavity method for sparse random graphs may be viewed as the macroscopic 
counterpart of belief propagation. If belief propagation converges, the cavity method 
describes its macroscopic statistics in the limit of infinite system size ITH] . A 
more careful interpretation is needed when belief propagation ceases to converge, and 
in this case more exotic statistical mechanics techniques can play a crucial role in 
generalizations of the belief propagation algorithm. 

When belief propagation (BP) breaks down, one may distinguish between different 
possible causes for this failure. One of these possible causes is replica symmetry breaking, 
related to the breaking of ergodicity. This is reflected by a free energy landscape 
exhibiting a structure of a large number of free energy valleys that are disconnected, in 
the sense that the correlations between them are smaller than correlations within one 
valley, and the free energy barriers between valleys are extensive. A generalisation of 
the cavity method on sparse graphs has been developed to take into account a first step 
of replica symmetry breaking [Tl] Ho] over the last years. Although this scheme was 
designed for the computation of macroscopic observables, i.e. averages over ensembles, 
it could be adapted to yield an algorithm that gives valuable new information on specific 
instances of hard optimization problems: survey propagation UK]. 

Survey propagation (SP) has proved to be a succesful algorithm which can cope 
with hard constraint satisfaction problems such as random 3-satisfiability and graph 
colouring . These problems become hard when the solution space breaks apart 
in disconnected components, such that the relative sizes of the basins of attraction 
corresponding to these solutions for algorithms like ordinary belief propagation vanish 
and many suboptimal fixed points appear. Loosely speaking, the basic idea of survey 
propagation is to gather the statistics of the fixed points that correspond to solutions 
of the constraint satisfaction problem i.. These "surveys" (probability distributions 
associated to each link of the graph) may be used for decimation of the original constraint 
satisfaction problem, until the problem is reduced to a sufficiently easy one that can 
be solved with a conventional algorithm. One of the reasons why SP is so succesful 
is the fact that it corresponds to a zero-temperature problem in statistical mechanics, 
having the consequence that the "surveys", which represent probability distributions, 
have a very efficient parametrization. This makes it possible to study very large graph 
instances. 

The 1 step replica symmetry broken (1RSB) version of the cavity method is 
applicable to finite temperature cases as well [Hj, suggesting a possible potential for 
the algorithmic counterpart on a specific instance. The benefit aimed at is that in 
cases where the free energy landscape is complex and belief propagation itself does 
not converge, the surveys provide sufficient information leading to improved marginal 

| Strictly speaking, there are a number of subtle differences, such as the assumption that the cavity 
fields take integer values. Note that nonconvergence of BP is not the only problem which is cured by 
SP: algorithms that are guaranteed to converge to local minima of the Bethe free energy in contrast to 
SP do not find solutions to 3-sat problems in the hard-sat phase ^E], i-e. the local minima one arrives 
at are not likely to be a groundstate 
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approximations. 

However, various difficulties, some practical and some technical, must be taken into 
account, which we will shortly list here, to be discussed more thoroughly later in the 
text: 

• The surveys we will be dealing with are not efficiently parametrized, since they can 
take values on a continuous interval. We will be forced to sample from distributions, 
which severely limits the system sizes. For infinite values of the connectivity of the 
corresponding graphs, more efficient representations become possible, as suggested 
in [HUHl]. 

• Given that we have a limited system size, we must remember that the approach is 
likely to be sensitive to finite size effects. One of the issues is the optimization of 
the free energy with respect to the replica symmetry breaking parameter m. This 
may be interpreted as zooming in at the states with the lowest free energy, where, 
moreover, the "complexity" (entropy of states) vanishes in a truly 1RSB scenario. 
It is questionable what remains of this idea for a small system size. 

• It is not straightforward to test the performance of the algorithm: if the output of 
the algorithm is a set of approximate single-spin marginal probabilities, how can 
we judge the quality of this approximation? Comparing to exact results is possible 
for very small system sizes only (N = O(10 2 )). 

• We should take into account the possibility of further steps of replica symmetry 
breaking, which might prevent the algorithm from converging, or cause malfunction 
in a different manner. 

The above complications show that our expectations regarding an algorithm based 
on the 1RSB generalisation of the cavity method should be modest. However, in 
the current paper we present encourageing results, indicating that there is at least a 
possible type of application, albeit for future generations of fast computers, in the field 
of error correcting codes. This point has very recently been recognized and suggested 
by Migliorini and Saad [21] for LDPC codes, where the authors argued that a 1RSB 
type algorithm might increase the dynamical limit of the channel noise for decoding. 
In this paper we for the first time propose such an algorithm for the biased finite 
connectivity Sourlas code with 3-spin interactions. We show how a finite temperature 
survey propagation algorithm gives accurate results in the regime where competing 
approaches fail to converge or fail to recover the encoded message. 

The suggested algorithm may be used for finite temperature decoding of encrypted 
messages, where the single spin marginals are the basis for inference of original message 
bit values. A corresponding performance measure of the algorithm is the bit error rate, 
which is easily calculated. Thereby the required knowledge of exact values of marginals 
is circumvented and one is not limited to small system sizes. The model we will study is 
a biased finite connectivity Sourlas code with 3-spin interactions. This model is closely 
related to a general 3-spin model with external fields. 
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In the next sections we will introduce the general model class of Ising 3-spin systems 
with external fields, we will shortly review belief propagation, the cavity method and 
it's 1RSB generalization, and finally describe the finite temperature generalization of 
survey propagation. Exact comparison of marginals is only possible for limited system 
sizes (N = 45 using a junction tree algorithm PQ), which indeed turned out to be too 
small to see any improvement in a few cases we studied. Therefore we consider, as a toy 
model, finite temperature decoding of the finite connectivity Sourlas error correcting 
code based on 3-spin interactions. As was also pointed out for the case of a CDMA 
model in [HH2D1, when decoding happens without the knowledge of the channel noise, 
one may suffer from replica symmetry breaking effects if the adopted estimate of the 
channel noise is below the Nishimori temperature [22]- The appearance of metastable 
states disrupts the behaviour of simple message passing algorithms before the onset of 
replica symmetry breaking of the equilibrium state |2Sj- As mentioned above, in j2Ij it 
was proposed that a 1RSB type algorithm might be able to increase the critical noise 
value for the decoding dynamics as compared to RS type decoding algorithms. 



2. The 3-spin model 

Although the formalism may be set up in a more general fashion, we will be focussing on 
systems of N binary variables Oi G { — 1,1}, i G {1,2,..., N} that interact on a random 
graph, where the order of the interaction is 3. The equilibrium microstate probability 
distribution will be described by the Boltzmann measure 

exp[-PH{tr)] 

P(<T) = E.exp[-^(r)] « 
where the Hamiltonian is given by 

v i 

where J v G {—1,1}. Greek letters fi, v will label interaction vertices (factors in a 
factor graph representation), z/j label spins in V{y\ the set of site indices contained in 
interaction v. We will be discussing random graphs with fixed connectivity, i.e. the 
number of interactions for spin i, \V(i)\, will be fixed and equal for all i, where V(i) is 
the set of all interactions containing i, i.e., we will choose = k Vi from now on. 

Consequently, the sum over v in the Hamiltonian (J2J) runs from 1 to kN/3, the total 
number of interactions. 

In a statistical inference problem, we assume the values of {Jv}-, and (3 are 
known, and the objective is the calculation of the partition sum, or of a marginalized 
probability, e.g. 

pfa) = X>(ir) (3) 
ar\i 

In general the calculation of this quantity requires 0(2^) operations, and thus for large 
iV one resorts to approximations. 
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In the regime of finite connectivity, the average loop length is of order log(iV), such 
that for large system sizes, correlations that propagate through loops may be neglected 
if the couplings are of order N° . In this regime the Bethe approximation, which is 
exact on tree graphs, performs well. Consequently, the cavity method gives rather 
accurate results, as long as the system is in a single state, and belief propagation is a 
good candidate for inference on a (large) single instance. [13] . For large values of (3 (low 
temperature values), replica symmetry breaking occurs, and a single state cavity method 
description is no longer accurate. Belief propagation ceases to converge, or gets trapped 
in a sub-optimal free energy minimum [2Sj, and does not give any information. In the 
absence of external fields, it has been shown that the 3-spin model with interactions 
G {—1, 1} is self-consistently described within the 1RSB cavity framework [21], for a 
fixed number of interactions per spin. 



3. The methods 

3.1. The cavity method 

The cavity method [22] E] i n it's simple form is based on two assumptions: 

• The values of the spins {o~j} that are neighbours on the graph of site i are only 
nontrivially correlated through the interaction v e V(j) H V(i) with site i. 

• In equilibrium, the system is in a single state (there is just one dominant free energy 
valley), i.e. there is a one to one correspondence between the states of neighbouring 
spins 

As a consequence of these assumptions, in the absence of their interactions v with spin 
0o, we may write the probabilities for 1(k — 1) neighbours {o~j} of cr as independent, 
parametrized in the following way: 

tffa) ~ exptfhfo) (4) 

Then linking the 2(k — 1) spins to spin 0, we may write 



(5) 



Po (°o) ~ ex P \ P a o( J2 [J v a U2 cr U3 + G Q )]) + f3 

{ CT v } { "ev(o) v 

It then follows that in this state the corresponding field ho associated to spin (which 
again misses an interaction with one other pair of spins, interaction fx) is given by 

h%= -rtanh-^tanh^J^tanh^Jtanh^J] +6 (6) 

On a sparse random graph with random interaction values, the macroscopic statistics 
will be governed by the distribution of cavity fields W(h) over the graph, which, due 
to the statistical independence of cavity spins, follows from iterating equation (JOJ in 
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distribution: 

k— 1 

W(h)= f HidhidgiWih^Wigi)} 
J i=i 

x { 6 { h ~ \ ?t tanh ^ tanh ^ ^ tanh (/^) tanh C%)] + 6 ^j ( 7 ) 

where the brackets denote averages over the interactions J and random external fields 
9. This equation may be solved numerically with a 'population dynamics' algorithm 
[Tlj . From the stationary distribution under this iteration algorithm, all macroscopic 
quantities can be derived. 

3.2. Belief propagation 

Belief propagation in essence is the above procedure applied to a specific instance of a 
(sparse) graph. Now the procedure is focussed on the computation of local quantities, 
all the cavity fields associated to the graph, i.e. one for each link. By iterating equation 
(0) for a given graph, one avoids averaging over the interaction values J and the external 
fields 9. If the algorithm converges to a fixed point, from the resulting cavity fields one 
may compute the single node "beliefs" 

k{o-i) = -(1 + mi(Ji) (8) 
which are approximations for the marginal probabilities 
bi(ai) ^p{o-i) 

(9) 

Their expressions in terms of the cavity fields are 

mi = tanh(/?#i) (10) 

where 

H i= £ ^tanh- 1 [tanh(/5J At )tanh(^ i )tanh(/?^ 2 )] + f (11) 

The 3-spin beliefs, approximations for the 3-spin marginals are computed from the cavity 
fields in a similar way. 

Belief propagation has been rediscovered several times, and has a close relation with the 
Bethe approximation. Fixed points of belief propagation are minima of the Bethe free 
energy, which is exact on tree-like graphs, but is often a good approximation when a 
graph contains loops [2o] . 

For lower temperatures and for graphs with interactions inducing frustration effects, 
the cavity method and belief propagation break down. Belief propagation often ceases 
to converge in these cases. One would expect a similar breakdown of the cavity method 
in this case, but since the order parameter of the cavity method is a macroscopic object, 
the behaviour is quite different generally. The distribution of cavity fields may still be 
stationary under the population dynamics iteration in parameter regimes where belief 
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propagation does not converge. However, this stationary distribution obviously does 
not represent the distribution of locally consistent beliefs. Thus the interpretation of 
this result is a more complicated issue in terms of local quantities§. It is suggested [13] 
that stability analysis through the iteration of perturbations in the distribution might 
reveal the location of the AT line (2Z|, at which the replica symmetric approximation 
becomes unstable. For the SK model j2H] on a dense graph this has been shown J3] 
to be the case indeed. Several heuristic or rigorous statements [33 IHJ] have been 
made concerning convergence of belief propagation in relation to the location in the 
phase diagram of Ising spin type systems. 

The cavity method has been extended to deal with cases in which replica symmetry 
breaking occurs, and thus it is worthwile to modify this method and apply it to the case 
of single instance statistical inference. In the same manner, survey propagation and its 
generalization for nonzero energy density [32] were developed for optimization problems. 
The main difference in the following discussion will be the application to problems at 
nonzero noise-level, i.e. at finite temperature. 

We will note here explicitly that this approach will not cure problems arising from 
short range correlations e.g. in regular lattices. A possible approach for this type of 
problems, which we shall not discuss here, is to take into account correlations between 
larger numbers of spins in the free energy. This idea was first introduced by Kikuchi 
[33] and leads to the cluster variation method [34 

3.3. One step replica symmetry breaking 

The cavity method at the level of one step replica symmetry breaking is usually applied 
at zero temperature [JJ3 EH 133 I3E] • The reason for this is that at T = on the one 
hand the equations simplify, and on the other hand there are more known applications 
in optimization problems for which a single instance generalization of this technique is 
fruitful (e.g. 3-sat, graph colouring). 

Here we will be applying the finite temperature version of the cavity method, 
of which we will just discuss the basic assumptions, and refer the reader to P3] and 
[To] for a more thorough description. The main assumptions, to be compared with 
the assumptions of the cavity method in it's simple (replica symmetric) form, are 
summarized in the following: 

In contrast with the 'single state' cavity method, one now assumes that an 
exponential number of states contribute to the Boltzmann measure. At each link of the 
graph, instead of regarding just one cavity field, one takes into account a distribution 
of cavity fields. The density of states at a given free energy is of an exponential form: 



where is the reference energy and m is the so-called replica symmetry breaking 
parameter. This assumption is shown to be consistent with the iteration of cavity field 

§ From a global point of view the equations are equivalent to the replica symmetric approximation of 
the macroscopic observables 




(12) 
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distributions, given that the value of the cavity field in a given state is uncorrelated 
with the free energy of that particular state. A consequence is that at each iteration 
the distribution is reweighted by a factor exp[— m/3AF], where AF is the difference 
between the free energies of the state before and after the iteration. Observables will 
first be summed over all states a weighted by their corresponding Boltzmann factors 
exp(— (3F a ) I J2j ex P( — flF 1 ), the result of which should be averaged over the distribution 
of free energies (fT2|). 

In practice, the calculation of observables is numerically implemented by an 
advanced population dynamics algorithm, in which N populations of M fields, each 
representing a distribution at a given link, are iterated. This procedure can be executed 
for all values of m £ [0,1], yielding a generalized free energy F(m). Selecting the 
equilibrium state is achieved by demanding dF(m) / dm = pK] . which is equivalent to 
zooming in at the free energy of states for which the entropy of states (or "complexity" ) 
becomes non-extensive [37]. 

The numerical cost of the procedure described above is considerable, since the cavity 
field distributions are represented in terms of (large) sample populations. Moreover, the 
extremization of the free energy with respect to m may difficult, when the variation of 
F(m) as a function of m is small. At zero temperature, for a large model class, the 
situation simplifies a lot, since the representation of cavity field distributions is much 
more efficient. In the case of discrete values for the local Hamiltonian (e.g. any spin 
glass with ±J interactions, k-sat energy terms) the cavity fields can be assumed to 
take integer values, such that the only degrees of freedom are coefficients of a small 
number of delta peaks. For random ^-satisfiability, a single number parametrizes the 
field distribution for each link on the graph. In the SAT phase, where the energy is zero, 
the reweighting of distributions by the free energy shift as described above effectively 
either leaves the distribution invariant, or rules out the state completely (contradictory 
states), leading to the very efficient algorithm of survey propagation on single instances 
of random graphs representing the satisfiability problem. 

4. Finite temperature survey propagation 

The goal of this paper in general is an investigation of the benefits of a survey 
propagation algorithm for finite temperature. The type of problem we address is 
the inference of marginal probabilities for a spin-glass type model displaying replica 
symmetry breaking effects. Along the lines of the one step RSB cavity method for finite 
temperature we iterate distributions of cavity fields, where instead of sampling external 
fields and interaction values at each iteration, we let the distributions evolve on a given 
instance of a sparse graph, thus fixing the value of 9 and J locally. For a given value of 
the replica symmetry breaking parameter, the iteration equations are the following: 





"£V(0) V 
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x<* { K - £ n tanh-^tanh^) tanh(/3/i^) tanh(^)] - 9 



(13) 



where 



AF({J„},{/£},{£})=]og2 + E log/ 



cosh(/3Jy) 



+ log cosh < (3 



v ^ 



and 



E u ( J "iKv9 

1 



^ev^ lcosh(/3it(J,,,/i* 

^0 



«(^, ^, <4) = « tanh- i [tanh(/5J,) taoh(^) tanh(/3<£ 2 )] 



(14) 



(15) 



From the distributions of cavity fields, one may construct distributions of "belief" fields: 



-mpt\F({J^MA,{9Z 2 }) 



MGV(O) 



X 



5 \Ho- E -^tanh-^tanh^tanh^Jtanh^)]-^ 



(16) 



where 



AF({JJ,{^J,{^J) = log2+ E !og{ 



cosh(/3J /J 



cosh(/3ti(J M , h^,gji 2 )) 



E ^,^,0 + ^0 

Lm£V(0) 

The marginal probability p(cro) for the value of the spin at site is now given by 



+ log cosh | (3 



p(o-o) = -[1 + ao I dH W (H ) tanh(/3if )] 



(17) 



Following [14] . we can deduce expressions for the free energy of the system F(m) and 
the derivative of the free energy dF{m) / dm. These are given in QAppendix A| ) 



5. The 3-spin model as a biased finite connectivity Sourlas-code for a 
binary symmetric channel 

The 3-spin model with finite connectivity may be regarded as an error correcting code of 
the Sourlas-type [SB] • The Sourlas code for p-spin interactions was originally studied for 
a fully connected model. In the limit p — > oo it is equivalent to the random energy model 
J3*H] and the Shannon limit is approached. The benefit of looking at finite connectivity 
codes and finite p is the fact that the transmission rate, which vanishes for the dense 
graph and p — > oo, is now finite [10]. However, the drawback is the fact that the bit error 
rate does not vanish, such that for practical purposes its potential is limited. Codes of 
the Gallagher type (low density parity check codes) |H] are more promising from that 
perspective, since they have vanishing bit error rate and approach the Shannon limit. 
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The optimal decoding temperature for error correcting codes can be shown to 
coincide with the temperature corresponding to the "channel noise" [22]- Since this 
temperature is equal to the so-called "Nishimori temperature" , it can be shown (under 
certain conditions, including the absence of external fields or "biases") that this 
temperature always corresponds to a replica symmetric phase in the phase diagram 
[12], suggesting that belief propagation may be an appropriate decoding algorithm. 

In real life, however, the channel noise might be an unknown quantity If the 
estimate of the channel noise in this case is too small, the decoding is likely to happen in 
a parameter region where RSB effects play a role, in which BP either does not converge, 
or gives bad results. One would expect that in this case, the 1RSB generalisation might 
be a robust alternative [23 d] 

A more detailed inspection of the generic behaviour of error correcting codes tells us 
that decoding problems are caused by the so-called dynamical phase transition [2H]: in a 
low temperature regime above some critical channel noise value p^, the fragmentation of 
the free energy landscape prevents the algorithm from finding the ferromagnetic state. 
Above an even larger noise value p c > p^, the free energy of a replica symmetry broken 
spin glass state is lower than the ferromagnetic free energy, such that decoding is not 
possible at all. Thus, the only region in which a 1RSB algorithm might improve the 
decoding performance, is in the region p^ < p < p c , thereby shifting the dynamical 
phase transition to a higher value. 

In the presence of external fields, the model may be interpreted as a biased 
Sourlas-code. The location in the phase diagram where improvement upon conventional 
decoding is to be expected, is close to the critical channel noise, for low decoding 
temperature. Since in the case of a random bias we might not have a sharp phase 
transition, we will have to be careful in the selection of a suitable parameter regime. 

In order to select a parameter regime in which our algorithm might perform better 
than BP, we will perform a short macroscopic analysis of the model. We will here shortly 
discuss the cavity analysis of the model for 3-spin interactions and random external 
fields. In (30] the model was studied for uniformly biased messages (a uniform external 
field) using the replica method. In principle, their result applies to the present case of 
random biases, due to gauge symmetry, for the appropriate choice of a message bias, 
but for clarity we will here discuss the derivation of this result to make the connection 
explicit. 

5.1. RS analysis of biased code for 3-spin interactions 

First of all, a locally biased message £ G { — 1, 1}^ is generated, according to 




(19) 



The external fields or biases will be generated from the distribution 




(20) 
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where we choose 

p{6i) = c(+)5(9 t -6) + c(-)6(6i + 6) + 0(0)5(6,) (21) 

with 9 constant and of course c(+) + c(— ) + c(0) = 1. 

Given the message, the code J e { — 1, l} Nk ' p is generated according to 

Note that the above choices correspond to a binary symmetric channel (BSC), for which 
each transmitted bit is likely to be flipped to its inverse with equal probability p c h given 
by 

/? P =^log^^ (23) 
2 Pch 

The original message is inferred according to 

i = sgn((«7 i ) /J ) (24) 
where the average is taken with respect to the Boltzmann measure 

exp[-/3H(tr)] 
J2 T exp[-(3H{T)\ 

Thus, for the finite temperature decoding at inverse temperature (5, the knowledge of 
the marginal p(crj) is required. Since we want to describe the macroscopic behaviour of 
the code, we use the cavity method to average over the disorder || (the messages £ and 
the interaction values J). 

Here we summarize the equations to be iterated using a population dynamics 
algorithm (see Appendix B for details) 

k—1 k—1 

W(h) = £ q{0 Q ) f J] [duiQimMh -5>- 6 a ] 



a " i=l i=l 

oPpJ 

Q{u) = E 



er 



j 2 cosh(/3 p ) 

dhdgW{h)W{g)5 



k k 

W{H) = Y,q{O a ) f \{[du t Q{u l )]5[H-Y,u l -6 a ] 

a J i=l i=l 

/2 = J dHW(H)sgn(H) (26) 

where 

Q(0a) = C(tt) (27) 

cosh^Ppfc'Q,) 

From these equations we compute the bit error rate p = (1 - /i)/2 as a function 
of the channel noise, for connectivity k = 4 at inverse temperature (5 = 5, where we 

|| Note that using the replica method in the replica symmetric framework one generally arrives at 
identical equations 
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Figure 1. Top: bit error rate p as a function of channel noise for a code with k = 4, 
[3 = 5 and bias 9 = 1.5 with ferromagnetic initial conditions versus spin-glass initial 
conditions. Bottom: The RS free energies corresponding to the top picture. In both 
figures the label F denotes ferromagnetic initial conditions and SG denotes the branch 
with spin-glass type initial conditions (see text). 

initialize the population dynamics algorithm with random positive fields uniformly on 
the interval < h < 3, or randomly on a symmetric interval —3 < h < 3. 

Until now, we did not discuss the need for the bias explicitly. It is well-known that 
an unbiased Sourlas code has such a small basin of attraction, that BP does not converge 
to the "ferromagnetic" or "retrieval" state for any order of the interaction larger than 
2, unless prior knowledge is taken into account in the initialisation [10] • Therefore, we 
have added random external fields, or random biases to a fraction of the sites, which 
have the effect of breaking the local symmetry. 

We chose the fraction of external fields according to the values c(0) = 0.8, 
c(+) = c(— ) = 0.1, and 9 = 1.5, such that the magnitude at low temperatures might 
compete with the local interaction terms at sites where 8^0. For a temperature 
sufficiently below the Nishimori temperature (/^Nishimori = P P ), i-e. (3 = 5, results are 
reported in figure ^ Clearly there is a jump in the bit error rate as a function of 
the channel noise around p c ^ = 0.028 for "spin glass" initial conditions, and around 



SG 
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Figure 2. Replica symmetric phase diagram of the finite connectivity Sourlas code 
for k = 4, p = 3, c(0) = 0.8, c(+) = c(-) = 0.1, 9 = 1.5. From top to bottom 
the lines indicate the Nishimori temperature, the dynamic: transition, the RS critical 
transition and the line along which the population dynamics algorithm ceases to find 
the ferromagnetic state. 

Pch = 0.78 for ferromagnetic initial conditions. We may interpret the former line as a 
dynamical phase transition for BP. Comparing the free energy values in both cases, we 
find that the replica symmetric approximation of the spin-glass free energy is nearly 
constant over a large interval, whereas the ferromagnetic free energy (which is expected 
to be replica symmetric indeed) is an increasing function up to the point where the 
population dynamics collapse to the spin-glass state. The RS critical channel noise 
corresponds to the point where the two lines cross, suggesting that retrieval is not at 
all possible above this point, around p^ = 0.065. Within the RS framework, therefore, 
the jump at p c h = 0.78 is not the physical critical channel noise. 

Figure El displays the location of the various transition points for a range of 
temperatures below the Nishimori temperature within the here described RS framework. 
Within a 1RSB framework the results are likely to change, since in general the spin- 
glass free energy will be higher than the RS approximated result. From the above RS 
analysis, we expect BP to fail for p c h above the dynamical transition line, and we expect 
improvement using the 1RSB treatment above the RS dynamical transition. 

6. Numerical results for single instance decoding 

We have tested a number of algorithms on single instances of a Sourlas code 
corresponding to the same parameter values as above, i.e. with (3 = 5, way below the 
Nishimori line in the critical region. Ordinary BP was able to find the ferromagnetic 
state roughly up to the value of p = 0.028, the point where the randomly initiated 
population dynamics algorithm ceases to converge. Beyond that point, BP ceased to 
converge too, even when we took into account the prior knowledge corresponding to the 
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external field in the initialisation. However, it turned out that a "damped" version of 
BP (i.e. at each timestep t + 1 replacing the field h t by (1 — e)h t + eh t+ i) improved the 
performance dramatically. In case this algorithm did not converge either, we applied a 
time averageing procedure, along the lines of The resulting algorithm was able 

to find an optimal ferromagnetic-type state (i.e. with bit error rate comparable to the 
lower branch in figure for nearly all instances with noise p c h < 0.07. 

For p c h = 0.07, we ran our 1RSB algorithm on 20 instances of a Sourlas code 
of iV = 999 bits (note that we need kN to be divisible by 3) and compared the 
results to those of damped time averaged BP (TABP). Additionally, we compared 
the results to those of a so-called "double loop" algorithm, which is guaranteed to 
converge to a local minimum of the Bethe free energy [33] %. For the sample size of 
the cavity field distributions we chose M = 1024, and we iterated for 50 updates per 
field distribution to get rid of transients before iterating another 50 updates for the 
calculation of observables. These numbers imply a considerable numerical cost, limiting 
the number of experiments we could perform in a reasonable amount of time. We ran the 
algorithm for the values m — 0, 0.05, 0.1, 0.15 and 0.2 of the replica symmetry breaking 
parameter. TABP was damped with e = 0.05, the resulting marginals were averages 
over the last 2500 (sequential) updates of 10000 in total. For each instance we repeated 
the latter procedure with random initial conditions 10 times. 

6.1. Results 

In figure 01 we report the results of the above experiments for p ch = 0.07 and p c ^ = 0.08. 
The results of the double loop algorithm, not surprisingly, clearly have the largest bit- 
error rate, since the algorithm is not designed to find a global minimum of the Bethe 
free energy. Generically, for small values of the replica symmetry breaking parameter 
m, the 1RSB algorithm settles in a suboptimal spin glass state, with a bit-error rate 
comparable to the double loop result. The resulting free energy, which increases with m, 
should be maximized with respect to m. However, before reaching a maximum, the free 
energy drops to the ferromagnetic free energy, the corresponding bit-error rate dropping 
simultaneously. The optimal value of m is selected by choosing the smallest m after the 
free energy drops. This behaviour was found for LDPC codes macroscopically in |2Tj . 
but was not yet applied to real instances. 

For p c h = 0.07 we observe that our algorithm finds the ferromagnetic state for each 
instance at some value of m, which still fluctuates between different instances due to 
finite size effects. We conclude that the damped TABP algorithm performs surprisingly 
well on some instances. However, whether it finds a near-optimal state depends on each 
individual run in general. The instances 1, 4, 7, 11, 15 and 19 all suffer from initialisation 
dependence and have small probability of convergence to the ferromagnetic state. The 

% Note that this algorithm is much more successful for regular lattices, where it provides an 
implementation of the cluster variation method. Here it is only used as a comparison since it will 
converge to a Bethe free energy minimum even when BP or damped BP does not 
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1RSB algorithm, in contrast, is robust in the sense that it finds the optimal state for 
some m, regardless of initial conditions. Note that the error-bars of the damped TABP 
algorithm are somewhat misleading, since they indicate the standard deviation, although 
the distribution of bit-error-rates is not symmetric. 

In figures |U we have plotted all values of the free energy as a function of the values 
of m for which we ran the algorithm, illustrating the behaviour described above. Note 
that the maximum of the spin glass free energy indeed seems to extrapolate to values 
higher than the approximate RS value (around —1.46, see figure HJ), indicating that the 
critical channel noise is likely to be increased in the 1RSB framework. 

For p c h = 0.08, the ferromagnetic state was not always found by the 1RSB 
algorithm, indicating either that this is above the 1RSB dynamical transition, or we 
should look in a larger range of m. However, when damped TABP finds a ferromagnetic 
state, in all cases the 1RSB algorithm finds it robustly. There was one exception to 
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Figure 4. Free energy densities as a function of the replica symmetry breaking 
parameter m for all instances at p c h = 0.07 



the latter point, in instance 6, where TABP found the ferromagnetic state in one of the 
ten runs. When we increased the value of m further, however, the algorithm stopped 
converging (i.e. the distributions of fields started fluctuating heavily). A further increase 
of the sampling size might be helpful here. 

Finally, to reduce finite size effects further, we ran the 1RSB algorithm for just 
3 instances for a large code of N = 4998 at the noise value p c h = 0.075. Results are 
reported in figures |SJ Still we observe different results for each instance, but again the 
1RSB algorithm finds the ferromagnetic state in all 3 cases. This result is spectacular 
in comparison to the performance of the TABP algorithm in the second instance , where 
it did not find the ferromagnetic state in any of the 10 runs. 

7. Discussion 

In this paper we have presented results of applying a finite temperature generalization 
of survey propagation on sparse graphs. This algorithm, though numerically expensive, 
seems particularly of interest for finite temperature decoding of error correcting codes 
at temperatures beneath the Nishimori line close to the critical channel noise. We have 
illustrated this by regarding a finite connectivity, partly biased Sourlas code with 3- 
spin interactions. As predicted in [21], the dynamical transition of belief propagation 
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of TABP at p c h = 0.075 and N = 4998. Right: corresponding free energy density as a 
function of m. 



is pushed towards larger values of the channel noise for temperatures beneath the 
Nishimori temperature. This feature is also reminiscent of zero temperature results 
obtained for survey propagation in the UNSAT phase [32] . 

For the model we studied, it turned out that a damped version of time averaged 
belief propagation performed particularly well too. However, the finite temperature 
survey propagation algorithm turned out to be more robust, and was able to find the 
retrieval state in cases where all other tested algorithms failed. 

We feel that this result is an encouragement to further investigate the properties, 
in spite of its numerical cost, of the algorithm, which should certainly be tested on the 
more realistic LDPC codes. 

Furthermore, the dependence of the performance on various parameters, like the 
temperature, the bias, and the system size should be investigated further. In principle, 
the simple phase space structure of unbiased error correcting codes at the Nishimori 
line suggests that in practice it should be easy to infer the value of the channel noise 
by varying the temperature in a BP algorithm alone, thereby circumventing the need 
for timeconsuming algorithms like the one proposed in this paper. The present analysis 
is therefore not intended as the basis for a practical algorithm, but rather as a proof of 
principle to show (indirectly) that a finite temperature 1 RSB method can accurately 
estimate marginal probabilities on problem instances where BP fails to converge. 

Accordingly, the correspondence to results of a macroscopic analysis of the biased 
Sourlas code within the 1RSB framework may be further specified, thereby more 
carefully specifying the regime in which finite temperature survey propagation would be 
an appropriate algorithm. The latter analysis was omitted in the present study because 
of its numerically extensive nature. 
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Appendix A. Free energy expressions in the 1RSB algorithm 

The expressions for the free energy F(m) and its derivative dF(m)/dm are generalized 
from in the same way as the iteration equations. Instead of a collection of N random 
samples of fields, constructed by sampling interaction values J and external fields 9 at 
each iteration, we now have a distribution of cavity fields explicitly associated to each 
link, which is iterated using the actual fixed value of J and 9 corresponding to the 
instance of the graph under consideration. 

Distributions of free energy contributions for k interactions in which a site is 
involved (AF^) and for each site linked to k interactions (AF- 2 ^) are constructed 
through the iteration equations 

-/3AF?-\{J (t ,h (tl ,h ll!t ,h (ta }) = log[cosh(/?J^)] 

+ log[l + tanh(/3J M ) tanh(/3/t w ) tanh(/?/i M2 ) tanh(/3/t M3 )] 
nev(i) 

cosh(/3J M ) 1 



-/3AFf ) ({J M ,V,^ 2 },^)=log2+ £ log/ 

M6V(i) I 

+ log cosh < (3 



cosh(/3w(J M ,/i M1 ,/i /i2 )) 



(A.l) 



u(J^, h^h^) + 9, t 

luev(i) 

The contribution of each site to the total free energy is (taking into account overcounting 
of link contributions) 

AFi = --AF 4 {1) + AFP (A.2) 
3 

in a replica symmetric setting. In the 1RSB framework, these quantities should be 
averaged over all states, weighted with their Boltzmann factors as in and this result 
is then averaged over the distribution of free energies. The resulting expressions that 
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are evaluated numerically are 



(A.3) 



with 



.(i) 



F (2) = 



1 



-mfSAF 



(l)c, 



(2)a 



m(3 M ^ 



(A.4) 



Appendix B. Cavity analysis of Sourlas code 

In the following it will be useful to introduce sublattices: I a = {i\6i = 6 a } to take 
into account the fact that the sites are not equivalent due to their biases. The order 
parameter will thus be the collection of three sublattice distributions of cavity fields, 
given the message: W + (h\£), W-(h\£) and Wo(h\^). The order parameter equations are 
given by 

k—i k—i 

w a (h\o = J n [duiQ a ( Ui \0]s[h -5><- e a ] 

i=l i=l 

q«ho= E c(a')cK)^Ef( j i^'np(r,e»") 

a', a" J 



x / dhdgW a ,(h\t')W a// (g\t")5[u - itanh- 1 [tanh(/3/i) taah(/3g) tanh(/?J)]] 



(B.l) 



or 



Qa(U ' e) = <£" C(a ' )c(a,/) ? § 8cosh(/3 p )cosh(/3A')cosh(/3 p ^) 

x / - ^tanh- 1 [tanh(/3/i) tanh(^) tanh(/3J)]] 

(B.2) 

These equations can be simplified by considering symmetries. First of all, we recognize 
that Q a (u\£) does not actually depend on a, and obeys Q{u\ — £) = Q(—u\£). From 
this symmetry it follows that W a (h\ — £) = W- a (— h\£). As a result we may restrict to 
considering either W a (h\+) or W a (h\— ). We will assume that c(+) = c(— ) from now 
on. This gives us 

e /3 P J 

Q(«I+) = E 



dhdg 



2cosh(/3 p ) 
c(0)W o (/i|+) + c(+) 



f3 p d 



cosh(/3 p 0) 



W+{h\- 



c - 



x 



c(o)^ (^l+) + c(+; 



cosh(/9 p ^) 



w + (0i+)+ C (-: 



cosh(/? p 0) 
cosh(/9 p ^) 



x (5 



u — — tanh 1 [tanh(/3/i) tanh(/3(?) tanh(/3 J)] 



(B.3) 
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We are interested in the overlap with the message at the site in question. Therefore we 
regard the joint distribution of field and message: 

k k 

W a (H,0 = p(^\O a )W a (H\0 = p(^\e a ) J f[[d Ui Q a (uM)}5[H - 9 a ] (B.4) 

1=1 1=1 

If the sign of ^ tanh(/5ifj) is positive, the message at site i will be decoded correctly. 
Thus the overall bit error rate is (1 — /i)/2, where 

» = \ E E c («) / dHW a (H, £)sgn(6ff) (B.5) 

Since p(£\-6 a ) =p(-£\9 a ) andW a (fT|-0 = W_ a (-#|0, we may deduce W a (H, -£) = 
Vy_ Q (— if, ^) and consequently 



c(0)W„(#|+) 

+ c (+) T^T ^+( g l+) + C H T7^T ^-( g l+) (B.6) 
cosh^Ppb 1 ) cosn^Ppfc 1 ) 

All equations clearly close in terms of Q(u), W(h) and W(H), where 



Q{u) = Q{u\+) 

W{h) = Y,Q(8a)W a (h\+) 

a 

W(H) = J2q(0a)W a (H\+) (B.7) 

a 

with the effective measure 
leading to equations (f2Tlj) . 
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